{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 21 - Simple linear regression\n",
    "\n",
    "We will use the dataset with information about the labor market for recent college graduates from Lab 20.  You can download the CSV file [here](http://comet.lehman.cuny.edu/owen/teaching/mat128/Feb2019_labor_market_majors.csv).\n",
    "\n",
    "We need to download another library called `statsmodel`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!pip install --user statsmodel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import Seaborn and the other libraries so we can use them in our code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import statsmodels.formula.api as smf\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading and cleaning the data\n",
    "\n",
    "As in Lab 20 we need to read our CSV file into a dataframe and clean it.\n",
    "\n",
    "Load the data in the dataframe `labor`, remembering to skip the non-data rows at the start and end of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display your `labor` dataframe below to check it was created properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make things easier, we will rename the columns to shorter names without spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labor.columns = [\"major\",\"unemployment\",\"underemployment\",\"early\",\"mid\",\"graduate\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we need to remove the commas from the `early` and `mid` columns, and change the column type to float, as in Lab 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "<code>labor[\"early\"] = labor[\"early\"].str.replace(\",\",\"\").astype(float)\n",
    "labor[\"mid\"] = labor[\"mid\"].str.replace(\",\",\"\").astype(float)</code>\n",
    "</details>\n",
    "\n",
    "Check that this code worked correctly by displaying `labor` again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple linear regression\n",
    "\n",
    "Recall from Lab 20 that the columns `early` (which was `Median Wage Early Career`) and `mid` (which was `Median Wage Mid-Career`) were the most correlated, with a correlation of 0.848. \n",
    "\n",
    "Let's remind ourselves what the relationship looked like by plotting a scatter plot with `early` on the x axis and `mid` on the y axis.\n",
    "\n",
    "<details> <summary>Hint:</summary>\n",
    "Scatter plot code pattern:\n",
    "<code>df.plot.scatter(x = \"column name 1\", y = \"column name 2\")</code>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can perform linear regression with the following code, which does not display anything:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lm = smf.ols(formula = 'mid ~ early', data = labor).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The formula is always `dependent_variable ~ independent_variable`, and in this form, the variable names, which are the column names, cannot have spaces in them or else we need to use `Q('variable name with spaces')`.\n",
    "\n",
    "All the information about the linear model is stored in the variable `lm`.  To see this information, type and run  `lm.summary()` below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a lot of information!  We will only focus on a few of the numbers.  The coefficients for the regression line can be found in the middle section, under the column `coef`: $\\beta_0 = 1.014 \\times 10^4 = 10140$ and $\\beta_1 = 1.37632$.  Thus, the regression line equation is $y = 1.37632x + 10140$\n",
    "\n",
    "You can also get the coefficients by typing `lm.params`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use Seaborn to plot the regression line on the scatterplot in one step.  Type and run the code: `sns.regplot(x = 'early',y = 'mid', data = labor)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To further check if the linear model is a good fit for the data, we can plot the residuals to see if they have a normal distribution.  To get the residuals, type `lm.resid` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you figure out how to plot the residuals as a histogram?  They are already stored as a Pandas Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the distribution of the residuals look normal?  Can you find the residual outlier on the scatter plot?  Can you figure out what major that is?  We looked at how to identify outliers in Lab 5. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "The outlier has the greatest mid-career median wage, so we can look for the row with the max value in this column:<br>\n",
    "<code>max_row = labor[\"mid\"].idxmax()</code><br>\n",
    "and then display that row:<br>\n",
    "<code>labor.iloc[max_row]</code>\n",
    "</details>\n",
    "\n",
    "### Regression line: underemployment vs. median wage early career\n",
    "\n",
    "The next two most correlated columns were `underemployment` and `early`, with a correlation of -0.564108. Compute the linear regression model with `underemployment` as the independent variable and `early` as the dependent variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display a summary of your linear regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the equation for the linear regression line?\n",
    "\n",
    "Use the Seaborn package to plot the scatter plot of underemployment vs. median wage early career with the regression line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the residuals as a histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is the distribution of the residuals roughly normal?  Is your linear regression model a good representation of the relationship between underemployment and median wage early career?\n",
    "\n",
    "### Challenges\n",
    "- In the green taxi trip data, what is the correlation between trip distance and fare amount?  Compute the linear regression line with trip distance as the independent variable and fare amount as the dependent variable.  Find the equation for the regression line, plot the scatterplot with the line, and check if the residuals are normally distributed."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}